Systems and methods for assisted surgical navigation

ABSTRACT

In at least one embodiment, a method of surgical navigation is provided. The method includes receiving an external three-dimensional model of a surgical site from the viewpoint of a headset, wherein the external three-dimensional model is derived from reflected light. The method further includes aligning the external three-dimensional model with an internal three-dimensional model of the surgical site from the viewpoint of the headset, wherein the internal three-dimensional model is derived from medical imaging, and generating an aligned view. The method further includes providing the aligned view to the headset, and updating the aligned view in real-time while the headset is moved or the surgical site is moved or modified during a surgical procedure.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 62/136,877, entitled SYSTEMS AND METHODS FOR ASSISTEDSURGICAL NAVIGATION, filed Mar. 23, 2015, which is hereby incorporatedby reference.

TECHNICAL FIELD

The embodiments described herein relate generally to systems and methodsfor computer-assisted surgical navigation.

BACKGROUND

Computer-assisted surgical navigation is at the threshold of arevolution in surgery by extending the surgeon's capacity to visualizethe underlying anatomy and by guiding positioning of instruments. Avariety of innovations in computerized virtual image collection,analysis, fusion and generation are driving these advances. Advanceshave been made in gaming hardware, military hardware, and augmentedreality by which direct pupillary projection is realized of imagescontaining a combination of a camera view and a virtual construct orconstructs. Increased numbers of procedures, such as implant placementin the hip and knee, can benefit from precise surgical navigation duringthe implantation. Improvements in outcomes, as by reduction in thenumber of revisions because of a misaligned implant, for example, wouldsave more than enough to warrant further investment in improved surgicalnavigation technologies.

For example, as known in the gaming arts, a physical motion sensor(typically a three-axis accelerometer or gyrosensor, more generally“inertial sensors”) can be combined with a camera and display, enablinga first person perspective through a visual window into a virtual spaceon a display, as is described in U.S. Pat. No. 8,913,009 to Nintendo.Relative spatiality is achieved by defining a stationary window. Thusfor example, a player may swing an actuator through the air in a virtualgolf game, causing a virtual ball represented on a viewing screen to flyas if struck. Representative patent literature describing the workingsof this technology includes U.S. Pat. Doc. Nos. 2012/0258796 and U.S.Pat. No. 8,100,769 to Nintendo, U.S. Pat. Nos. 6,285,379 and 8,537,231to Philips, and related art for interactive virtual modeling hardwareand software such as U.S. Pat. Doc. Nos. 2005/0026703 to Fukawa,2009/0305785 to Microsoft, and U.S. Pat. No. 7,705,830 to Apple and U.S.Pat. No. 7,696,980 to Logitech, which disclose technologies fordispensing with keyboards in favor of haptic gesture sets and resultantrepresentation and control of interactive processes. Depth modeling ofphysical objects using a structured pattern of dots generated byinfrared emitters is described in U.S. Pat. Doc. No. 2014/0016113 toMicrosoft and in U.S. Pat. No. 6,891,518 to Siemens.

Surgical use is known in the art. U.S. Pat. Nos. 6,787,750 and 6,919,867to Siemens describe use of optical fiducials to measure depth andlocation in a surgery. In U.S. Pat. No. 6,919,867, an example is given(Col 4, line 8-Col 6 Line 42) where a surgeon is provided with a view ofinternal anatomical structures through a head-mounted display whileoperating. A correct anatomical orientation relative to the patient'sbody is achieved by mounting retroreflective optical beacons on thepatient and around in the workspace and by employing image analysis toidentify the location of the beacons. Computing means are taught forrelating a coordinate system associated with the camera with acoordinate system relative to the patient's body, and for tracking thecamera as it moves with the head of the surgeon. However, after almosttwo decades of development, the resultant systems utilize cumbersomeretroreflective balls that must be fastened to bones and surgical toolsso that their positions can be mapped, and any images in the headsetdisplay appear superimposed on nearfield elements such as the surgeon'shands, defeating the surgeon's hand-eye coordination. As a result, mostsurgeons have reverted to display of the virtual images on a remotedisplay that is accessed by looking up and away from the surgical site.

Infrared markers have also been used for dental surgery (Hassfeld, S etal. 1995. Intraoperative navigation in oral and maxillofacial surgery.Intl J Oral Max Surg 24:111-19). Correlation between CT and patient skinsurfaces for guiding surgical procedures was achieved using a laserscanning system (Marmulla R and Niederdellman H. 1998. Computer-assistedbone navigation. J. Craniomaxillofac Surg 26:347-59) and later by thesame group (Markerless laser registration in image-guided oral andmaxillofacial surgery, J Oral Maxillofac Surg 62:845-51). However, thesesystems required immobilization of the patient in a reference framedevice and again use a remote display to present the image synthesis soas to avoid visual illusions that are paradoxical and confusing.

All these systems rely on optical image analysis that depends on cameraframe grabbers that are inoperable and blind when a needed line of sightis blocked. Optical systems are not operative when lighting isinsufficient or a direct optical path to the target is obstructed orunrecognizable, such as when smeared with blood or when a surgeon'shands or a surgical instrument is blocking the view from the camera.Image analysis to recognize and triangulate optical fiducials is alsocomputationally intensive, which can be slow or halting, and has theeffect of limiting the availability of computer assisted surgicalnavigation systems by driving up the price and increasing systemcomplexity.

Early computer-aided operating systems include HipNav, OrthoPilot andPraxim. Technologies of relevance have been developed by Simbionix, 3DSystems, BlueBelt Technologies, Medtronic and Siemens. But disadvantagesof computer-assisted surgery remain. A major disadvantage is cost, whichis generally prohibitive for many hospitals and surgery centers.Improvements have added to the cost, not reduced it. The size of thesystems is also disadvantageous. Large C-arms or O-arms and windows takeup space in the surgical suite, an important disadvantage in alreadycrowded operating rooms of modern hospitals or clinics in that theequipment becomes a liability when fast action is needed and access isimpaired. Additionally, another disadvantage of most surgical navigationsystems in current use is the need for intraoperative computerizedtomography (CT) imaging, which exposes the patient and staff tosignificant doses of ionizing radiation.

As applied to surgery, conventional systems generally use a collectionof retroreflective spheres that serve as fiducial markers. Clusters ofspheres are attached to surgical instruments so that orientation anddepth can be monitored using cameras. A pattern of infrared dots isprojected onto the surgical field and analysis of the centroid of eachdot on spherical surface permits acquisition of the position of eachfiducial. Each surgical instrument must include at least four fiducialmarkers for complete orientational mapping and the needed resolution ofthe centroids requires a fairly large tetrahedral cluster be used.Fiducial clusters may also be attached to the patient, such as byclipping the marker to an exposed bone. These reflective spheres are notuseful, of course, if the optical path is blocked, as occurs frequentlyin surgery during the more invasive parts of the procedures.

Optics for infrared wavelengths rely on illumination outside the rangeof human vision, and hence have been adopted as a foundationaltechnology. However, the technology may be better suited to inanimateobjects rather than warm bodies. Dichroic mirrors and bandpass filterswill not readily separate broadly emitting objects in the 700 to 1200 nmrange. Surgical lamps, reflections of hot bulbs off chrome steel, andtools such as cauterizing tips may cause spurious images and add tocomputation time.

Binocular visors are known in the art and may be used in place of aremote display screen. However, by blinding the surgeon to all butcamera generated views, the surgeon can be no more perceptive than thecapacity of the system to generate a lifelike display in the visor.Surgeons wishing to rely on an unaided eye and their own hands toperform the procedure must remove the visor. The difficulty of thisunsolved problem is underlined in recent literature reports (BichlmeierC and N Navab, Virtual window for improved depth perception in medicalAR; Blum T et al. 2012 Mirracle: an augmented reality magic mirrorsystem for anatomy education. IEEE Virtual Reality).

Moreover, a difficult challenge has not been solved, that of presentingthe fusion data as a virtual image that appears as the surgeon would seeit in first-person perspective, dynamic and moving with the position ofthe physician's head and eyes so as to have a believable sense of depth,where the skin and the surgeon's hands are superimposed above the deeperstructures. Advantageously, the view would appear as if the surgeon wasprovided with the capacity to look beneath the skin or surgical fieldand see underlying boney and visceral structures beneath. The surgicalnavigation tool would take on a compact and wearable format, such as amonocular eyepiece affixed to a headset worn to the operating room bythe surgeon. In order to use this as an interactive intraoperativetechnique, a library store of patient imaging data must be fused withthe surgeon's visual perspective of the surgical field so that a virtualfusion image is presented in correct anatomical alignment andregistration. By so doing, the improved imaging modality can haverelevance to and can be validated by the surgeon's inherent sense ofspatial location, anatomy and general surgical know-to-do derived fromyears of visual, tactile and kinesthetic sensory experience. The imagingmodality thereby would also avoid a need for cumbersome patientregistration frames and remote display systems.

Also desirable is a system enabled to segregate elements of the visualfield. In a first embodiment, segregation is done to identify individualbones in a dataset derived from tomography or from an AP and Lateralview by X-ray. The individual bones or clusters of bones may then beprojected into a synthetic virtual view according to their surgicalrelevance. It then becomes possible to isolate the bones from thepatient and to do more detailed analysis of structure of individualbones and functional interactions between small sets of bones.Segmentation also includes computer power to isolate visual elementssuch as the hands and fingers of the surgeon, surgical tools andprosthetics while reducing virtual clutter. Surprisingly, when this isdone, any relevant virtual elements of the patient's anatomy and avirtual database segmenting the surgeon's hands may be operatedcooperatively to show the hands occluding the virtual anatomy—or avirtual pair of hands operating in an enhanced virtual space. These andother inventive systems have not been realized in the art and are anobject of the invention and is difficult or impossible to achieve usinglight-based image analysis and optical fiducials at any wavelength.

Thus, there is a need in the art for an intraoperative three-dimensionalvirtual viewing system that overcomes the above challenges, isperceptually integrated into the surgeon's view of the operation inprogress, includes both haptic and pre-haptic interfaces, and overcomessystem blindness when line-of-sight is blocked. Depth-enhanced virtualviews of any surgical instruments and prosthetics manipulated by thesurgeon are also desirable for making measurements of angles andguidepaths on instrumental approach to a surgical target, such as inimplantation of surgical fixators or replacement joints, for example. Anovel approach to these and other issues facing modern surgery isdescribed that surprisingly is computationally simple and fast and hasbeen enhanced to rely on the surgeon's touch and gestures as well asvirtual image display, thus providing essentially a multi-sensorialextension of the surgeon's senses in integrated computer-assistedsurgical navigation systems and methods.

SUMMARY

In at least one embodiment, a method of surgical navigation may includereceiving an external three-dimensional model of a surgical site fromthe viewpoint of a headset, wherein the external three-dimensional modelis derived from reflected light. The method may further include aligningthe external three-dimensional model with an internal three-dimensionalmodel of the surgical site from the viewpoint of the headset, whereinthe internal three-dimensional model is derived from medical imaging,and generating an aligned view. The method may further include providingthe aligned view to the headset, and updating the aligned view inreal-time while the headset is moved or the surgical site is moved ormodified during a surgical procedure.

Surgical medicine can benefit from whole a new generation of informationtechnology advances, particularly in virtual imaging. The embodimentsdisclosed here are driven by an ever-increasing demand to reduce patientcosts and risks, improve patient safety, efficiency, and surgicaloutcomes. However, development of realistic virtual surgery systems forinvasive surgical procedures remains one of the most challengingproblems in the field of virtual reality based surgery (and surgicaltraining) because of the complexity of anatomical structures, theirchanges in pathological states, and the need for detailed informationabout surgical tools and prosthetics used intraoperatively. While notgenerally cited, the surgeon's hands should also be considered in anycomprehensive answer to the problem, both because they are frequently anobstruction to visual interrogation of the surgical field and becausetheir motion and any gestures made offers information that can informthe system display. When used in combination with a segmented library ofanatomical parts, tools and prosthetics, the capacity to also segmentthe surgeon's hands offers multiple advantages in reducing imageclutter, improving depth cues, and directing computing operationswithout interference from background noise and without the need forremote control interfaces.

While not generally appreciated, the surgeon has the capacity tointegrate augmented imagery presented to a single eye with a nativevisual field presented to an unaided eye. Integration involves thecorpus callosum and optic chiasma in the brain, which are neurologicallyintegrated with motor functions in both hemispheres. Thus, embodimentsmay be designed to take better advantage of this inherent ‘wetware’ bybetter aligning the surgeon's pupillary view in the unaided eye with theaugmented virtual elements presented through a monocular or headset. Afaster image refresh rate and attention to vanishing point geometry inraytrace software, along with high fidelity optical pathways, may beused to achieve the coordination whereby effortless inter-hemisphericcoordination of hand-eye motion is realized.

In an embodiment, the surgeon may be wearing a headset having aneyepiece, a camera for collecting reflected light, a projector forprojecting an array of light beams onto a surgical field, and aneyepiece projector or optronics element for providing the virtual imageonto or through the eyepiece and into the pupil, wherein the headsetincludes a digital connection to a computing machine having at least oneprocessor, at least one memory for storing the computerizedtomographical scan, and programming instructions for constructing theexternal three-dimensional model from optical data received by thecamera and for constructing the virtual image derived from thecomputerized tomographical scan (or other imaging modality) according toanatomical points of reference detected in the externalthree-dimensional model. The computing machine may also include aco-processor or a server for generating and analyzing internal andexternal three-dimensional wireframe models.

The external three-dimensional model may be aligned with a plurality ofanatomically correlated emission sources, such as an active radiobeacon,a reflective RFID tag, any radio reflector, or an optical beacon thatare enabled to continue to provide orientation information even if thesurgical site is blocked by an arm, a surgical instrument, or a machinesuch as a C-arm.

Surgical instruments may also be tracked, each instrument being modifiedto emit a signal indicative of a location relative to the external planeof the surgical field. The surgical instrument and eyepiece may beoperated cooperatively to display and/or stream numerical data such asdepth, angle, relative angle, relative elevation, volume, temperature,pressure, or a more specialized sensor output, such as oxygenation orenervation.

In another embodiment, an umbilical connection to a computing machineand a dot matrix projector is provided so as to relieve the surgeon froma larger headpiece weight. The digital connection may comprise a bundleof optical fibers, and the computing machine may be a server digitallyconnected to the headset by the optical fibers.

In another embodiment, the surgeon may be enabled to select a part ofthe virtual image by pointing at the part with a laser pointer, andraise the part away from the surgical field for closer inspection. Thepart may be manipulated by rotation and magnification according to handgestures as a virtual image projected into the eyepiece. Software may beused to provide a reference library model from which views of a patientvolume can be obtained from any depth and any angle. Individual bones oranatomical elements may be selected for inspection in the virtual fieldabove the surgical site or in situ, including temporal sequences showinga series of surgical events from a surgical plan.

This embodiment may use software to construct 3D models from tomographicdatasets and to segment out individual anatomical elements such asbones, and optionally soft tissue features such as organs, nerve tractsand ligaments. Segmentation can be computationally intense and may bedone offline before starting the surgical procedure. Segmentation may beperformed by a process of comparing datasets with reference datasets onhuman anatomy and may be confirmed by teaching. Prior to operating, asurgeon may indicate the relative anatomy and may educate the system bypointing to each anatomic element in turn and naming it.

Suitable libraries of segmented images of the patient's anatomy may bestored in computer memory for use during a surgical procedure. Theinternal images may be acquired by computerized tomography (CT), MRI, orother imaging modalities, for example, while not limited thereto.

Radio signals may be used to supplement the digital mapping and forupdating relative alignment and orientation so as to speed the initialfusion and any update required when there has been a break in the visualmap continuity. Map elements may be lost when optical data streaming isinterrupted, such as by turning the headset away from the surgicalfield, or by putting a hand on an incision site, and so forth.

Processing the digital data may include performing triangulation basedon the one or more acquired signals and a distance relationship betweena transmitter that outputs the one or more emitted signals and areceiver that receives the one or more reflected signals. The system maybe optically frameless and patient registration may be achieved by aninternal to external mapping correlation routine that is directed by thesurgeon so that the external wireframe is fused to the solid model ofthe underlying anatomy. Subsequent updates may be tracked by monitoringthe position of the headset, either inertially or with reference toradiobeacons. Individual beacons may be passive reflectors and may beconfigured to reflect a signal that has an identifiable signature so asto speed acquisition of the general orientation and alignment of thecoordinate systems. The radio system may supplement the optic system andallow all the data sets to be brought into a common frame of reference.Advantageously, radiobeacons may be placed at the corners of a Mayotable, a slip-on cover on the Mayo table, the corners of the operatingtable, or a mat under the surgeon's feet, each corner having a radioreflective antenna equipped with an identifiable signature reflection.In this way, the headset orientation may be tracked by an externalreference frame, but one that is not subject to the weaknesses ofoptical tracking. The surgeon may calibrate the system by pointing outat least one beacon associated with a boney prominence or obviousanatomical feature that is present on the wireframe map and the internalsolid model and the rest of the beacons can then be formed into aspatial map that is determinate for the duration of the procedure. Ifthe patient is rolled over, for example, only one or two beacons aredisturbed, so their positions may be refreshed while the remainingbeacons may be fixed. Tracking the headset may use standard matrixtrigonometry and require substantially less computational power.

Alternatively, active radiobeacons may be used, each emitting an encodedidentifier. Time of flight (TOF) measurements may be utilized asdescribed here to map each beacon relative to a stable externalreference frame achieved by tracking radiobeacons embedded in a surgicaldrape over the surgical site or positioned on a Mayo table or at thecorners of an operating gurney. By determining the distance to an activeradiobeacon from several radio receivers, the location of the beaconrelative to the reference frame may be accurately determined. Theseprinciples can be realized using active or passive radiobeacons.

In another embodiment, a separate optical system may be used to trackthe pupil and lens curvature of the unaided eye, and an algorithm may beemployed to derive a vanishing point that correctly renders the virtualinformation presented to the augmented eye. In this way, the brain isoffered information having sufficient visual depth clues that motorcoordination may be informed by the augmented virtual information. Forexample, the surgeon may not have to look up to read graphicalinformation presented in the augmentation. Data streams may appear tofloat near to, but not impede, the unaided eye's point of focus.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the drawings and the followingdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

In the detailed description that follows, embodiments are described asillustrations only since various changes and modifications will becomeapparent to those skilled in the art from the following detaileddescription. The use of the same reference numbers in different figuresindicates similar or identical items.

FIG. 1 shows an example system by which at least aspects of surgicalnavigation may be implemented;

FIG. 2 shows a frontal view of an example headset by which at leastaspects of surgical navigation may be implemented;

FIG. 3 shows a side view of an example headset by which at least aspectsof surgical navigation may be implemented;

FIG. 4 shows an example headset view of a surgical field by which atleast aspects of surgical navigation may be implemented;

FIG. 5A shows an example mapping system for generating athree-dimensional external model of a surgical field by which at leastaspects of surgical navigation may be implemented;

FIG. 5B shows an example data transformation by triangulation togenerate an array of Cartesian datapoints;

FIG. 6 shows an example process flow for generating a three-dimensionalvirtual fusion view by which at least aspects of surgical navigation maybe implemented;

FIG. 7 shows an example process flow for updating a virtual fusion viewby which at least aspects of surgical navigation may be implemented;

FIG. 8A shows an example mapping system for generating athree-dimensional external model of a surgical field by which at leastaspects of surgical navigation may be implemented;

FIG. 8B shows an example data transformation by triangulation togenerate an array of polar data points;

FIG. 9 shows a cross-sectional view of an example umbilicus to a headsetby which at least aspects of surgical navigation may be implemented;

FIG. 10 shows an example dot array by which at least aspects of surgicalnavigation may be implemented;

FIG. 11 shows an example structured dot array by which at least aspectsof surgical navigation may be implemented;

FIG. 12 shows an example dot array by which at least aspects of surgicalnavigation may be implemented;

FIG. 13A shows an example snap wireframe model by which at least aspectsof surgical navigation may be implemented;

FIG. 13B shows an example tertiary wireframe model by which at leastaspects of surgical navigation may be implemented;

FIG. 14A shows an example headset view having a polar grid for mapping asurgical field by which at least aspects of surgical navigation may beimplemented;

FIG. 14B shows an example virtual fusion model by which at least aspectsof surgical navigation may be implemented;

FIGS. 15A-15C show an example strobe sequence by which at least aspectsof surgical navigation may be implemented;

FIGS. 16A and 16B show an example process flow for building athree-dimensional virtual fusion view by which at least aspects ofsurgical navigation may be implemented;

FIG. 17 shows a block diagram of an example computing system by which atleast aspects of surgical navigation may be implemented;

FIG. 18 shows an example headset view of a select command by which atleast aspects of surgical navigation may be implemented;

FIG. 19 shows an example headset view of a levitate command by which atleast aspects of surgical navigation may be implemented;

FIG. 20 shows an example headset view of a rotate command by which atleast aspects of surgical navigation may be implemented;

FIG. 21 shows an example headset view of a zoom command by which atleast aspects of surgical navigation may be implemented;

FIG. 22 shows an example headset view of an angle measurement command bywhich at least aspects of surgical navigation may be implemented;

FIG. 23 shows an example headset view of a slice command by which atleast aspects of surgical navigation may be implemented;

FIG. 24 shows an example headset view of surgical instrument positionalanalysis by which at least aspects of surgical navigation may beimplemented;

FIG. 25A shows an example unaided view of surgical gloves;

FIG. 25B shows an example headset view of a visible surgical glovemanipulating a virtual object;

FIG. 25C shows an example headset view of an invisible surgical glovemanipulating a virtual object;

FIG. 26 shows a schematic representation of an example operating roomhaving projection systems by which at least aspects of surgicalnavigation may be implemented;

FIG. 27 shows an example headset view of multiple virtual objects bywhich at least aspects of surgical navigation may be implemented;

FIGS. 28A and 28B show a schematic representation of an examplefingertip portion of a surgical glove by which at least aspects ofsurgical navigation may be implemented; and

FIG. 29 shows an example radiobeacon reference coordinate frame andheadset by which at least aspects of surgical navigation may beimplemented, all arranged in accordance with at least some embodimentsdescribed herein.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented herein. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe Figures, can be arranged, substituted, combined, separated, anddesigned in a wide variety of different configurations, all of which areexplicitly contemplated herein.

Certain terms are used throughout the following detailed description torefer to particular features, steps or components, and are used as termsof description and not of limitation. As one skilled in the art willappreciate, different persons may refer to the same feature, step orcomponent by different names. Components, steps or features that differin name but not in structure, function or action are consideredequivalent and not distinguishable, and may be substituted hereinwithout departure from the present disclosure. Certain meanings aredefined here as intended by the inventors, i.e., they are intrinsicmeanings. Unless otherwise defined, all technical and scientific termsused herein have the same meaning as commonly understood by one ofordinary skill in the art to which this disclosure belongs. In case ofconflict, the present specification, including definitions, willcontrol. The following definitions supplement those set forth elsewherein this specification.

“Computer” refers to a virtual or physical computing machine thataccepts information in digital or similar form and manipulates it for aspecific result based on a sequence of instructions. “Computing machine”is used in a broad sense, and may include logic circuitry having aprocessor, programmable memory or firmware, random access memory, andgenerally one or more ports to I/O devices such as a graphical userinterface, a pointer, a keypad, a sensor, imaging circuitry, a radio orwired communications link, and so forth. One or more processors may beintegrated into the display, sensor and communications modules of anapparatus of the invention, and may communicate with othermicroprocessors or with a network via wireless or wired connectionsknown to those skilled in the art. Processors are generally supported bystatic (programmable) and dynamic memory, a timing clock or clocks, anddigital input and outputs as well as one or more communicationsprotocols. Computers are frequently formed into networks, and networksof computers, including servers, may be referred to here by the term“computing machine.” In one instance, informal internet networks knownin the art as “cloud computing” may be functionally equivalent computingmachines, for example.

“Server” refers to a software engine or a computing machine on whichthat software engine runs, and provides a service or services to aclient software program running on the same computer or on othercomputers distributed over a network. A client software programtypically provides a user interface and performs some or all of theprocessing on data or files received from the server, but the servertypically maintains the data and files and processes the data requests.A “client-server model” divides processing between clients and servers,and refers to an architecture of the system that can be co-localized ona single computing machine or can be distributed throughout a network ora cloud.

“Processor” refers to a digital device that accepts information indigital form and manipulates it for a specific result based on asequence of programmed instructions. Processors are used as parts ofdigital circuits generally including a clock, random access memory andnon-volatile memory (containing programming instructions), and mayinterface with other digital devices or with analog devices through I/Oports, for example.

“Software” may be described in the general context of computerexecutable instructions, such as program modules, being executed by oneor more computers, such as client workstations, servers or otherdevices. Those skilled in the art will appreciate that computer systemshave a variety of configurations and protocols that can be used tocommunicate data, and thus, no particular configuration or protocol isconsidered limiting.

“Data fusion” refers to the process of integration of multiple data andknowledge representing the same real-world object into a consistent,accurate, and useful representation.

“Segmentation” relates to image analysis in which individual structuralelements in a three-dimensional image are abstracted from the image andindividually modeled. Once modeled, those elements may be manipulatedindependently.

“Jitter” refers to the level of variation in a clock frequency percycle.

“Sampling rate” refers to the number of measurements made per intervalof time.

“Synchronous upsampling” as applied here relates to extrapolating asmooth measurement from a stepwise digital measurement by continuouslyevaluating a bracket of measurements with a slight lag from a real-timedata acquisition rate.

“Bit depth” indicates the level of resolution in a binary digitalmeasurement scale.

“Arthrospatial” relates to the spatial disposition of anatomicalfeatures in a solid model, particularly applying to boney structures.

“Polar Coordinate system” refers to a spatial mapping system having afixed centerpoint point (analogous to the origin of a Cartesian system)called the “pole”, where the ray from the pole in the fixed direction isthe polar axis. The distance from the pole is called the radialcoordinate or radius, and the angle is the angular coordinate, polarangle, or azimuth. In three-dimensions, a “z” depth is also used todefine the position of a point in an array relative to the pole.

“Surgical navigation” as used here relates to a method for conducting asurgical procedure using augmented views of the surgical field, oftools, of prosthetics, or of the surgeon's hands, including a virtualmodel of patient anatomy, preferably with segmentation of individualanatomical elements. The position of the tip of an instrument, forexample, may be conveyed to the surgeon by an imaging system (i.e., asystem that relies on transmission or reflection of an applied energy tocalculate the position of the tip relative to the anatomy). Machinefeedback may also be incorporated and used as a complement to humansenses of sight and touch as used to guide surgery.

“User interface” refers to a feature of a computing system configured toconvert a user signal such as a selection or a gesture into a machinecommand or a response to a machine request for input.

“Haptic” refers to the quality of a user interface enabled both todisplay images and to respond to touch commands applied to theinterface. Haptic commands can be applied to the surface using a fingeron a capacitive, inductive, pressure or temperature-sensitive panel orscreen. The term “tactile” refers to the sense of touch, while thebroader “haptic” encompasses both touch and kinesthetic information, ora sense of position, direction, motion and force.

“Pre-haptic” is used to denote a user interface in which gestures infree space are used to command execution of computer-driven routines.Gesture control may include a joystick on a gaming console, a button ona machine, a virtual “soft” button on a capacitive or inductive panel orscreen, a laser pointer, a remote pointer, a mouse or keyboard forcontrolling with cursor, and also verbal commands, while not limitedthereto. Pre-haptic commands can also include arm or finger motions as avocabulary of gestures recognized by an interface camera or an inertialsensor, for example. A combination of a pre-haptic and a hapticinterface is also conceived here.

“Stereopsis” refers to the perception of depth and three-dimensionalstructure obtained on the basis of visual information deriving from twoeyes by individuals with normally developed binocular vision. Illusionsof stereopsis may be simulated using raytrace software for creating atwo-dimensional perspective view in a monocular such that theperspective is a convincing representation of a scene having the neededvanishing points and other visual clues consistent with a depth of fieldhaving good correspondence between the visual perception and motorfeedback obtained by reaching into the visual field.

“Palmar” is used to describe the densely enervated anterior side of thehand, including the palm, fingers and fingertips, while “dorsal” is usedto describe the back of the hand. The hand generally begins at thedistal end of the wrist joint defined by the radius and ulna. The palmaraspect of the hand includes the dermis, an underlying palmar aponeurosisattached to the dermis by minute fasciculi, and underlying nerve rootsand tendons.

General connection terms including, but not limited to, “connected,”“attached,” “conjoined,” “secured,” and “affixed” are not meant to belimiting, such that structures so “associated” may have more than oneway of being associated. “Fluidly connected” indicates a connection forconveying a fluid therethrough.

Relative terms should be construed as such. For example, the term“front” is meant to be relative to the term “back,” the term “upper” ismeant to be relative to the term “lower,” the term “vertical” is meantto be relative to the term “horizontal,” the term “top” is meant to berelative to the term “bottom,” and the term “inside” is meant to berelative to the term “outside,” and so forth. Unless specifically statedotherwise, the terms “first,” “second,” “third,” and “fourth” are meantsolely for purposes of designation and not for order or for limitation.Reference to “one embodiment,” “an embodiment,” or an “aspect,” meansthat a particular feature, structure, step, combination orcharacteristic described in connection with the embodiment or aspect isincluded in at least one realization of the present disclosure. Thus,the appearances of the phrases “in one embodiment” or “in an embodiment”in various places throughout this specification are not necessarily allreferring to the same embodiment and may apply to multiple embodiments.Furthermore, particular features, structures, or characteristics of thepresent disclosure may be combined in any suitable manner in one or moreembodiments.

Referring to the figures, FIG. 1 shows an example system 100 by which atleast aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. A headset105 worn by a surgeon may include a projection system for displaying avirtual view of a solid model of internal bones and organs in aneyepiece. The projection system may be driven by a computing environmentthat may be local or remote, for example including an umbilicus 110 totransmit data from a remote computing machine with processor, memory andprogram-specific instructions. Alternatively, computing hardware may bebelt mounted or even miniaturized for mounting as part of the headset105. In a preferred embodiment, data exchange may be conducted to andfrom the headset 105 through a databus optionally including fiber opticsin the umbilicus 110. The headset 105 may also include a light patternprojector for casting an array of dots, either as a random speckledpattern or as a structured array, onto the surgical field 115. Thispattern is then captured by a pair of cameras with frame grabbersmounted so that different angular views taken at a single instant may beused to triangulate the position and elevation of unique dotsidentifiable in both captive images. Reflected light specific bandpassfilters may be used to reduce false signals from ambient light; furtherspecificity may be obtained in brightly lighted environments by strobingthe excitatory projector.

Once a wireframe map of the surgical field 115 is obtained, anatomicalreference points may be identified by image analysis or may be assignedby an operator, and a solid model may be oriented and aligned so thatthe anatomical features not visible beneath the exterior view of thesurgical field 115 may be matched and projected in the virtual viewdisplayed in the eyepiece. The solid model data may be acquired fromcomputerized tomography (CT) scans, magnetic resonance imaging (MRI), orother scans already of record in the patient's digital chart. In ageneral computational approach, thin slices may be merged to generate asolid model, and the model may be digitally segmented to isolateindividual anatomical structures such as bones, vessels, organs and thelike.

The computing environment provides the resources to do a data fusion ofthe two image datasets (the external view and the solid model) and tocontinuously update this according to the viewpoint of the eyepiece,which may be updated by inertial guidance or in reference to beaconsplaced around the margins of the field of view, for example, so that asimple reverse triangulation provides the spatial location and elevationof the eyepiece from the reference beacons. These beacons may be opticalor radiobeacons, for example, the radiobeacons enabling updated trackingof the solid model even when a visual obstruction or lack of lightblocks the camera view. Advantageously, the system can thus continue tooperate for short periods even in complete blackness, such as in theevent of a power failure, if provided with backup power.

FIG. 2 shows a schematic frontal view of the headset 105 of examplesystem 100 of FIG. 1, arranged in accordance with at least someembodiments described herein. Shown from left to right are: apod-mounted camera 200 on a headframe 205 that is worn over the eyes.Above the camera 200 is a small diode laser pointer 210 that is keyed toidentify where the camera 200 is pointed and may be used for adjustment.The head frame 205 may include a nose bridge and right and left lenssupporting arms. A head strap 215 may be used to secure the headset 105in place. Center top is an infra-red dot projector 220 serving to paintthe surgical field with a fine pattern of dots. As shown here, a singleeyepiece 225 is worn. The eyepiece 225 operates with an eyepieceprojector 230 to display a virtual solid model view of the surgicalfield to the eye of the wearer. Alternatively, the eyepiece 225 mayextend across both eyes of the user. A second camera 235 with a laserpointer 240 is mounted on the opposite side of the head frame 205. Bothcameras may be slaved to frame grabbers either by CCD or CMOS chips thatare synchronized to capture an image of the dot pattern at a preciseinstant of time. The cameras may be slaved to a master clock so that dotpulse rate is coincident with image capture or the projector 220 may beoperated continuously and both cameras operated with an adjustable butsimultaneous frame capture rate. Raytrace software may be used to imparta generally stereoscopic view of the surgical field and underlyingstructures. The intent is to provide the user with an image that appearsto be a three-dimensional representation of the underlying anatomy, suchthat it validates the surgeon's inherent sense of spatial location,anatomy and surgical know-to-do derived from visual, tactile andkinesthetic senses. All visual features may be presented as superimposedon the patient body form in correct anatomical alignment andregistration.

Methods for projecting a realistic virtual view continue to improve.Recent art includes US Pat. Publ. No. 2015/016777 to Magic Leap, whichdescribes planar waveguides, piezo drive units, vibrating opticalfibers, Bragg gratings, and other improvements of optoelectric eyewearthat enable the presentation of a virtual image to a viewer as ifprojected in real space and can include eye tracking. Also of relevanceare U.S. patent Ser. Nos. 13/915,530 and 14/205,126, all of which arehereby incorporated by reference.

A radio emitter 245 is shown centerwise at the brow of the headset andmay be used in conjunction with radiobeacons, RFID chips or other radioreflectors as described in more detail below to aid and guide inregistration of the virtual image with the eyepiece display 225. Radioreceivers 250, 255, 260 having defined spatial geometry may be mountedon the outside ends of the headset 105. Using laser beams, LIDAR mayalso be used to construct a radiometric landscape of the surgical field.The imaging system thereby overcomes the need for cumbersome patientregistration frames, reflective fiducials, and remote display systems.

FIG. 3 shows a schematic side view of the headset 105 of example system100 of FIG. 1, arranged in accordance with at least some embodimentsdescribed herein. In this view, the headset 105 is supplied with lenses300 that may work cooperatively with the eyepiece 225 to aid indisplaying three-dimensional “see into” images to the retina of one eyeor both eyes.

Also shown here is the umbilicus 110 for data transfer to and from theheadset 105 to a remote computing environment. Computational tasks maybe divided between local functions of a headset microcomputer andcontroller, and a workstation or networked processing environmentoperating over distributed workstations, either inside or outside theoperating suite.

Large amounts of data may be needed in addition to any camera imagescaptured by the headset 105. A CT solid model typically consists ofmillimeter-thick sections that have been fused using a pixel-basedextrapolation process into a solid model.

Surgical navigation of the present disclosure relies on an accuratethree-dimensional solid model of the patient. Data libraries having thisdata may be acquired through a number of medical imaging technologiesincluding CT, MRI, x-rays, quantitative ultrasound scans, and so forth.Scans using a variety of methods, such as CT and MRI can sometimes becombined with other datasets through data fusion techniques. Theobjective is the creation of a three-dimensional solid model thataccurately depicts the anatomical volume under the surgical field. Ofthe available scanning methods, a primary CT model is preferred becauseMRI data sets may have volumetric deformations that may lead toinaccuracies. For example, a data set may include the collection of datacompiled with 200 CT slices that are 1 mm apart, each having megapixeldensity. Pixel contrast provides sufficient detail of soft versus hardtissue structures to allow a computer to differentiate and visuallyseparate the different tissues and structures. The model may then beregistered with anatomical landmarks in the surgical field so that anyprojected virtual image in the eyepiece 225 is correctly registered withrespect to the relative positions of the patient and the surgeon, and isthen updated in real-time to maintain this correctness. Large memoryresources and algorithms for image correlation may be used to make fineadjustments in the display presented to the surgeon.

Where needed, the solid model may be supplemented by reference anatomymodels, such as for reconstructive surgery and for supplementing CTscans with standard soft tissue landmarks.

In a preferred embodiment, the surgeon may light up the referencelandmarks using a laser beam, pointing at significant reference featuresand stating an anatomical name that is recognized by the system, so thatthe surgeon essentially teaches the system the mapping points to be usedin stabilizing the images so as to smooth real-time scanning as theheadset 105 is moved, and so as to prevent loss of coherence whenobjects such as the surgeon's hands obstruct the solid model view andthe external view.

The resulting virtual model constructs and integrates patient-specific,anatomically correct, and comprehensive three-dimensional models, withany soft and hard tissue details available. Advantageously, having thisinformation allows the surgeon to highlight a particular feature, suchas an L3 lumbar vertebra, and abstract it above the surgical field forcloser inspection. The solid model of the vertebra may be rotated andsectioned, angles may be measured, and the vertebra reinserted into thespine of the patient in a virtual space, complete with an intended trackto be followed by a drill, and information sufficient to identify thesize of screw that is optimal, for example. Calling out the L3 or T12vertebra may be done with a laser pointer, a pointing finger, or averbal command, for example.

Data concerning dimensions and angles may be displayed on the eyepiece225 and verbal commands may be used in conjunction with a laser pointerto project lines through the body and measure cross-sectional areas, forexample. Measurement data are generally transmitted to a remotecomputing environment, and display views may be transmitted back to theheadset 105. Several devices for projecting a virtual view onto aneyepiece are known in the art.

FIG. 4 shows an example headset view 400 of a surgical field by which atleast aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. The headsetview 400 shows an incision 405 and a virtual CT model 410 of underlyingboney structures in anatomical registration.

Shown in this example is T12, the last thoracic vertebra, and a sectionof exposed lumbar vertebra as would be operatively exposed to repair afractured pedicle or facet joint. Each vertebra may be identified by aspinous process 415 that is visible through surrounding fascia. Ifneeded, a tool with a radiobeacon or light emitting diode may be used toidentify the exposed anatomical features in the surgical view 400 as afirst step in aligning a CT model with the patient as supine on anoperating table. With increased sophistication, image analysis may beused to identify the relevant anatomical landmarks automatically.

Shown are the lumbar vertebrae and a partial view of the sacrumadjoining the iliac crest. Radiobeacons may be placed as markers toaccelerate re-alignment of the display when the surgeon blocks thevisual view with an arm or looks away.

Surgical instruments 420 may also be tracked, such as for apparent depthof penetration. Each instrument is generally logged into a library ofinstruments stored in a computer memory, and position, alignment andregistration may be done continuously during the procedure so that notools are left behind.

FIG. 5A shows an example mapping system 500 for generating athree-dimensional external model of a surgical field by which at leastaspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. In thisexample, a depth map may be constructed from a generally random patternof speckled light 515 painted onto the surgical field by a projector505. A collimating lens 510 may be used to extend the coherency of thespots to a useful distance. A detailed three-dimensional externalwireframe model of the surgical field may then be constructed fromimages captured by two cameras 520 and 525 at a single instant but fromdifferent angles. FIG. 5B shows an example data transformation bytriangulation to generate an array of Cartesian datapoints having depthand position.

The apparatus of FIG. 5A utilizes an illuminated-pattern projector 505for painting the surgical field and a pair of cameras 520 and 525 forcapturing the dot pattern as projected. As can be readily appreciated,the projector 505 may include one or more light emitting diodes (LEDs),or a higher powered laser with a beam splitter, which may transmit thelight through a lens and/or other optical train to produce a desired dotpattern output. The size of the dot array is scaleable, and the level ofdetail achieved is dependent on the collimating lens 510 used to projectcoherent light onto the subject. With proper attention to detail,submillimeter resolution is possible. As the surgeon moves closer to thesubject, the dot pattern projected from a head-mounted system becomesmore dense, so that resolution is a function of distance.

The cameras 520 and 525 may each include a CMOS (ComplementaryMetal-Oxide Semiconductor) or CCD (charge-coupled device) sensor forcollecting a pixelated image of the surgical field. Note that in FIGS. 1and 2, the dot matrix projector and cameras are shown as visible fromthe front view, although they may actually only be visible through anintervening component such as a lens and/or filter.

For example, bandpass filters 530 and 535 may be used to filter outundesirable signals such as from ambient light or body heat. For noisereduction, a relatively narrow slice of the far infrared wavelengths(e.g., 1050-1150 nanometers) may be used, as may be generated from aminiature YAG laser. One way to make the images generally robust againstsources of interference such as sunlight is to use the bandpass filters530 and 535 in conjunction with digital rolling shutters that aresynchronized with strobing of the LED projector 505. Strobing in generalallows higher processing speed, as well as reduced energy consumptionand limits the need for cooling of the electronics.

In general, the IR projector 505 may be a single emitter that transmitsan optically-focused pattern including a plurality of spots or “dots”that optically populate the field of view. The projector 505 maytransmit IR light via optics, such as through a multi-lens array, adiffraction grating, and/or prismatic or Fresnel-based technology, whichcreates a pattern of a plurality of well-defined light spots.Alternatively, multiple light sources may be used, and indeed, thisallows for different, per-spot parameters such as timing, intensity, andother unique encoding signatures that facilitate individual spotcorrelation and pattern recognition.

A pair of light-sensitive cameras 520 and 525 placed off-axis from thetransmitter acquires any reflected spot pattern from a reflectivesurface within range. For example, the reflected dot pattern may begathered by a focusing lens in the receiving element onto the surface ofthe sensor's imager and captured by a frame grabber so that the twoframes (one from each camera) are captured at a single instant of time.Alternatively, a single light-sensitive camera (having appropriatefilters) placed off-axis from the transmitter acquires any reflectedspot pattern from a reflective surface within range. For example, thereflected dot pattern may be gathered by a focusing lens onto thesurface of the sensor's imager and a more computationally intensiveprocess may be used to calculate spot positions and a depth maptherefrom.

To this end, because the baseline separation of the two cameras (orbetween the dot matrix projector and the camera) is known, atriangulation algorithm may be used to determine depth and position. Oneor more spots in the projected pattern allow for computation of adistance result, e.g., as in the top view of FIG. 5A, where the bodysurface represents a reflective surface, the solid lines represent thetransmitted beams and the dashed lines represent reflections incident onthe camera lens. Even more spots in the projected pattern allow thedetection of a change in the reflective entity's elevation and/ororientation, as in the simplified two-dimensional view of FIG. 5B wherethe sensor detects an example target T*. The actual implementation maybe more complex because the headset may be tilted on multiple axes.

Where blue, purple, violet or ultraviolet light is used to generate adot matrix pattern, corresponding camera optics and bandpass filters areneeded to acquire images suitable for depth and position mapping.

A processor with sufficient volatile memory and clock speed may run analgorithm or set of algorithms to calculate the geometric offsets ofeach spot, e.g., based upon its centroid. Along with a distance, achange in floor elevation and/or surface orientation may be computed.

The distance calculation is generally invariant to the spot intensityand is based upon digital data so as to be less susceptible to analoginterference. The dot matrix projector 505 may be dynamically adaptiveto provide intensity adjustment according to the ambient lighting,surface properties, and the required sensitivity. For example, when dotsare output onto a highly reflective surface, less intensity may beoutput, and conversely more intensity may be output for a surface suchas skin that does not reflect particularly well. Any suitable frame ratemay be used depending on the application, e.g., 15 to 500 frames persecond, or even higher, with a suitable camera selected based upon theneeded/desired frame rate. A typical frame rate may be about 50 fps, butthe faster the frame rate, the less latency, such as for obstacledetection, and the more data is available for processing (e.g., ifneeded to discard poor map quality images). The timing may be such thatthe beam output is pulsed, with background correction being made basedon background intensity between pulses. A chopper may also be used.

A signature may be encoded into the dot matrix signal, e.g., viapulsing, to further provide robustness. In this way, for example, areflected signal received at an allowed frequency and/or at the correctsynchronized time, but that does not have the correct signature, may berejected as likely being from interference.

The detected distance may be used for obstruction detection, forexample. The geometry and/or displacement of each spot may be used inthe computation. Note that in a situation where no reflection is sensed,beacon inputs may be used to determine whether the system is faulted orthe cameras are simply not pointed at the surgical field.

Lasers may include diode pumped YAG lasers that advantageously may beobtained in a range of sizes down to about 1 cm³ and that may havewell-defined homogeneous frequency output at about 1080 nanometers.Doubled neodymium YLF/YAG lasers are also preferred for their small sizeand low cost. Q-switched microlasers are well suited for strobingapplications although somewhat larger in size. DFB lasers are tunable,and may be useful in defining spot signatures, and are very small.Alternatively, blue, violet and UV lasers may be used such as describedin U.S. Pat. No. 6,002,695. Typically, UVA and UVB emissions associatedwith tissue damage are avoided, but some violet and long UV wavelengthshave been associated with skin wound healing.

Fiber coupling permits larger laser packages to be projected onto thesurgical field via an umbilicus as shown in FIGS. 1 and 3. DPY lasersalso may be investigated if desired. Thus the laser package is readilyobtained from commercial sources and may be provided with a lens systemsuitable for generating a random dot pattern or a structured lightpattern as will be described below.

Blue, violet and UV lasers include Nitrogen lasers, excimer lasers,metal-vapor lasers, but more generally may include purple lasers such asKrypton and Argon excitation, GaN laser, and certain dye lasers.Reflected light and fluorescent emissions may be enhanced with filters.

Alternatively, analyzing the geometric movement of spots, e.g., byprocessing to find the centroid, provides one means of analyzing spotdata to produce a depth map with a single camera, although this demandshigher levels of computing power. Having multiple independent spotsprovides redundancy and robustness of the imaging system. By encodingcertain spots, patterns may be more readily recognized, speeding theprocess of mapping.

FIG. 6 shows an example process flow 600 for generating athree-dimensional virtual fusion view by which at least aspects ofsurgical navigation may be implemented, arranged in accordance with atleast some embodiments described herein. In Step 1 of this embodiment,two cameras are used to draw a correlation map based on images of spotspainted on the target by a suitable dot projector system.

In Step 2, the correlation mapping assumes no particular geometry ofspots and may be a goodness-of-fit difference map of spot centroids orpairing in the two images. The correlation model may consist of datafields having a position and an elevation and may be used to draw awireframe model from which anatomical features may be identified.

In Step 3, the wireframe model and a reference 3D solid model (such asfrom a CT scan) may then be processed by data fusion processing as knownin the art to produce a virtual solid model, termed here a “fusion 3Dmodel” extending from the surface of the surgical field to any internalstructures observable in the CT scan, as correctly registered accordingto the body position and observable anatomical features that werecaptured in the earlier step. If needed, beacons may be used to assistin registration, or an instruction subroutine may be run where a surgeonpoints out the relevant registration guides on the external wireframemodel and in the CT solid model so that the computer may propagate thealignment and generate virtual views of the model for projection intothe eyepiece, where the images may be further lensed if needed so as tobe correctly portrayed on the retina of the wearer.

The process is iterative. FIG. 7 shows an example process flow 700 forupdating a virtual fusion view by which at least aspects of surgicalnavigation may be implemented, arranged in accordance with at least someembodiments described herein. Several stages of each iteration areidentified in FIG. 7. Process flow 700 may include various operations,functions, or actions as illustrated by one or more of blocks 705, 710,715, 720, 725 and/or 730. Process flow 700 may begin at block 705.

Block 705 may refer to pulsing a dot pattern on the surgical field.Block 710 may refer to identifying and then mapping reflections (anylight returning to a camera lens). Block 715 may refer to identifyinganatomical landmarks in a wireframe model of the surgical field. Block720 may refer to fusing the dataset with a three-dimensional solidmodel. Block 725 may refer to using the resulting “fusion 3D model” togenerate virtual images for projection onto the eyepiece of the headset.Block 730 may refer to updating the eyepiece position and angulation.Process flow 700 may be repeated at a speed sufficient to assist thesurgeon and validate surgical intuition and direct observation duringthe surgical procedure. In advanced steps, surgical instrumentation mayalso be tracked, and geometric calculations needed to orient andproperly perform surgical steps may be shown in virtual tracks projectedonto the virtual fusion image.

In one instance, a probe may be manipulated to penetrate the bodyanatomy under the direct observation of the surgeon. The surgeon retainsa view of the surgical site and can directly control the implementaccording to best judgment, using the added three-dimensional view as anaid in successfully completing the procedure with a higher level ofaccuracy and confidence.

As needed, individual bones or anatomical features may be highlightedand zoomed in for a closer look, or picked up and raised into a virtualspace above the surgical field, where they can be studied in rotation orsection, picking out cross-sectional areas, lengths, widths, depths,fractures, bone density measurements, and the like so as to validatesurgical judgments as to the best approach to the task at hand.Interferences may also be assessed and anatomy may be heuristicallycompared with reference models to assess reconstruction where, forexample, complex fracture and dislocation has occurred. The condition ofsoft tissue may also be studied by superimposing additional scans suchas an MRI in path view. The approach is flexible and scaleable, allowingthe surgeon to augment the power of a basic structural fusion view(combining an external wireframe with an internal CT solid model) withan MRI solid model or ultrasound Doppler views, for example.

The position and alignment of the headset cameras or radio receivers maybe fixed and known relative to the position and alignment of theheadset's emitters. As a result, the change in the geometry informationof the illumination spot (or laser highlight) may be algorithmicallycalculated by comparing frame captures from each of the two sensors (orby comparing time of flight for radio chirps) to produce an accuratedistance to a reflective entity (e.g., an object or surface) within thesurgical field of view. Use of unstructured and structured light to mapsurfaces are described in US Pat. Publ. Nos. 2013/0100256 and2014/0016113. These publications are hereby incorporated by reference.

One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

In an illustrative embodiment, any of the operations, processes, etc.described herein can be implemented as computer-readable instructionsstored on a computer-readable medium. The computer-readable instructionscan be executed by a processor of a mobile unit, a network element,and/or any other computing device.

FIG. 8A shows an example mapping system 800 for generating athree-dimensional external model of a surgical field using structuredlight by which at least aspects of surgical navigation may beimplemented, arranged in accordance with at least some embodimentsdescribed herein. In this embodiment, an array of fiber optics is usedto convey structured arrays of pulsed spots to a collimating lens 810,which projects the light onto the surgical field. Laser diodecollimators for laser pattern generators are known in the art. In theseviews, the dot matrix projector 805 is mounted on the headset 105,however, an overarching C-arm, a chest mount, or a ceiling mount mayprovide advantages in some operating theaters. An alternate projectormay also be available. And by using supplemental beacons, obstructionsto the dot pattern may be tolerated by continuing to monitor position ata radio frequency and relying on the computational power of the systemto make accurate extrapolations of the underlying anatomy.

In another embodiment, three-dimensional models of implants may beembedded in a computer library, so that fitting of an implement may besolved by the computer and presented in virtual chronology or“guidepath” that the surgeon may elect to follow when placing theimplant. Any interferences and incorrectness of implant size (as wouldbe experienced post-operatively by impingement or ligamentousinsufficiency) may be avoided, leading to better outcomes.

FIG. 8B shows an example data transformation by triangulation togenerate an array of polar datapoints having depth and position.Structured Illumination has the advantage that a polar coordinate systemand zero reference point are inherent in the imaging projector and canbe used to speed analysis of the remaining mesh map. Data analysis isillustrated schematically in FIG. 8B and generates a dataset havingposition coordinates and elevation coordinates that can be used to drawa wireframe model and describe a body volume that can then besupplemented with data from one or more internal three-dimensional solidmodel scans. The combined “fusion 3D model” may then be used to generateprojections of virtual images that may be transmitted or piped into theeyepiece for focusing on the retina of the wearer, along with anynumerical data or streaming numerical data of relevance to the surgicalprocedure. Thus, for example, proximity to an artery can be detected bypulsatile pressure on a probe and nerve function can be assessedelectrically, extending the range of the surgeon's senses beyond thevisual and tactile.

FIG. 9 shows a cross-sectional view of an example umbilicus 900 to aheadset by which at least aspects of surgical navigation may beimplemented, arranged in accordance with at least some embodimentsdescribed herein. In this example, the umbilicus 900 provides a fiberoptic array 905, a databus 910, and power 915 to the headset 105. Fiberoptic bundles 905 may include subarrays 920 of fibers 925 by thethousands as needed to individually paint a surgical field with readilyencoded spots. Use of encoded spots may increase hardware costs but mayresult in significant increases in calculation efficiency and speed,reducing latency in the image refresh rate. A shielded databus 910 isalso shown, and while having only an 8-bit bus size (plus parity bit) inthis example, may be scaled to include larger parallel bus bundles witha baud rate limited only by the clock frequency of the processors. Fiberoptics may also be used for data transmission at GHz speeds, improvingsystem refresh rates and resolution.

FIG. 10 shows an example dot array 1000 by which at least aspects ofsurgical navigation may be implemented, arranged in accordance with atleast some embodiments described herein. In this example, the dot array1000 may have 60 degree sectoring and a center dot used for rapidlysolving the geometry of the spot array and generating a wireframe map.

FIG. 11 shows an example structured dot array 1100 by which at leastaspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. In thisexample, the dot array 1100 may be scaleable for detailed mapping of asurgical field. The spots may be regularly spaced to improve the speedof geometrical calculations. A wireframe model solution generallypropagates from a plurality of known reference points such as the uniquehexagon in the center of the field, and leads to coalescence of theentire dot field from multiple centers, as may be derivedcomputationally by using one or more co-processors or parallel processornodes by computational technologies known in the art.

FIG. 12 shows an example dot array 1200 by which at least aspects ofsurgical navigation may be implemented, arranged in accordance with atleast some embodiments described herein. In this example, the dot array1200 may have a dot subset for calculating a snap wire frame model. Athigh speed, a first-pass coarse wireframe model may be constructed fromthe index dots 1205 (black centers). Then a finishing step may beperformed using a more dense dot pattern 1210 (open circles).

A high-speed, large broadband databus may be preferred, but the softwarefor manipulation of the solid model and projection of realistic virtualimages is designed to reduce the computational workload using structuredlight in one embodiment. Speckled light may be used in otherembodiments, consisting essentially of generally irregular dot patternsprojected by imprecise optical trains, but those skilled in the art willrecognize that a corresponding increase in computational load may resultdue to the need for error correction and more probabilistic algorithmsfor assigning datapoints in building comparative maps from twosynchronous camera frames taken from different angles. Use of a dualframegrabber and software capable of assigning dot pairs x₂y₂z₂) fromthe two frames based on frequency encoding may accelerate the neededtriangulation to construct a depth map and correlate that with a livecamera view. Subsequent processing may be done by raytrace software orequivalent means for generating a fusion virtual image that may beprojected onto the eyepiece with a selectable level of transparency ofthe virtual features. The user can select a fully transparent view ofthe solid model, a view that is transparent only where a surgicalincision is made (or about to be made), or an opaque view that showsonly the external image.

Advantageously, by using a cluster of laser emitters in combination witha beam splitter that divides the output among multiple fiber opticstrands, patterns of spots having multiple frequencies can be formedinto a quilt, where the frequencies may be encoded to acceleratesolution of the three-dimensional geometry. Once the dot array issolved, motion of the headset may be tracked using accelerometry ormotion relative to fixed reference points, and the dot array may berefreshed accordingly so that raytracing of the virtual image is alwaysaligned according to the perspective of the user and is anatomicallycorrect.

The software can learn reference points defined by the user, either bymarking those spots with a laser pen, or using radiobeacons placed forexample at major boney crests. The software can also recognizeunderlying fascia surrounding the spinous process of each vertebralbody, for example, after a first incision is made and the cut retracted,so that the three-dimensional solid model may be correctly alignedrelative to the patient's anatomy on the operating table. Otheranatomical reference points are well known to physicians and may beentered using a combination of a voice identifier and a laser pointer,for example. Alternatively, RFID pads may be adhered to anatomicallandmarks surrounding the surgical site, providing a means to align thesolid model with the patient's position on the table even when visualcues are temporarily blocked.

FIG. 13A shows an example snap wireframe model 1300 by which at leastaspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. Similarly,FIG. 13B shows an example tertiary wireframe model 1305 by which atleast aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. FIGS. 13Aand 13B together represent steps for creating the snap wire frame model1300 and building the detailed model 1305 from the skeletal array of thefirst step. The propagation of the fine mesh model may occur frommultiple centers and may coalesce in a complete map. Advantageously,areas of the map that are unchanged may be bypassed when refreshing thefusion three-dimensional solid model and downstream virtual imagesstreamed to the headset.

FIG. 14A shows an example headset view 1400 having a polar grid formapping a surgical field by which at least aspects of surgicalnavigation may be implemented, arranged in accordance with at least someembodiments described herein. In this example, the headset view 1400provides a conceptual polar grid for rapidly mapping a surgical fieldand for building a three-dimensional fusion model incorporating CT dataas projected into virtual images displayed in the eyepiece of theheadset 105.

As shown in FIG. 14B, the resultant three-dimensional fusion model 1405is analogous to “X-ray vision” and allows the surgeon to visuallyinspect, for example, the overlying tissues, incision, and theunderlying vertebra, shown here as an open window to the lumbar spine.As before, radiobeacons or optical beacons may be used to supplement theview and improve the speed of image re-integration followinginterruptions in the data stream such as by movement of an arm throughthe field or a glance up to the wall clock. But then, the eyepiece mayalso include a display of a time indicator, so the physician is lesslikely to be distracted during critical steps of the operation.

FIGS. 15A-15C show example views of a strobe sequence whereby dot datamay be presented to the processor in sectors. Beacons may be used to addsupplemental confirmatory arthrospatial datapoints. RFID chips or hybridreflector/radio tags may also be used. Tags having prismatic coloredsurfaces may also serve as unique arthrospatial locators that supplementthe patterned projection.

FIGS. 16A and 16B show a two-page example process flow 1600 for buildinga three-dimensional virtual fusion view by which at least aspects ofsurgical navigation may be implemented, arranged in accordance with atleast some embodiments described herein. In this example, dataprocessing steps are shown for rapidly building a detailedthree-dimensional virtual model starting from a snap wireframe of a datasubset (Step 1). In Step 2, two frame grabbers are used to capture aninstantaneous view of an encoded dot pattern that is pulsed on asurgical field. The cameras may be synched to the projector and betweenpulses acquire native RGB views that can be used to enhance the qualityof the virtual images and to subtract background light. In Step 3, thetwo camera frames are used to draw a correlation map based on images ofthe spots; more than one filter may be used to selectively acquirecertain frequencies in rapid succession so as to collect the encodedspot data. Each frequency set helps to rapidly construct a coherentpattern across the entire surgical field. In Step 4, a correlation modelis assembled, the model consisting of a database having data fields,each data field having a position and an elevation that may be used todraw a wireframe or “mesh” model from which anatomical features and abody solid outline may be identified. In Step 5, the wireframe model anda reference 3D solid model (such as from a CT scan) may then beprocessed by data fusion processing as known in the art to produce avirtual solid model, termed here a “fusion 3D model” that extends fromthe surface of the surgical field to any internal structures observablein the CT scan, as correctly registered according to the body positionand observable anatomical features that were captured in the earlierstep. If needed, beacons may be used to assist in registration, or aninstruction subroutine may be run where a surgeon points out therelevant registration guides on the external wireframe model and in theCT solid model so that the computer may propagate the alignment andgenerate virtual views of the model for projection into the eyepiece(Step 6), where the images may be further lensed if needed so as to becorrectly portrayed on the retina of the wearer.

The process is iterative. As shown in FIG. 16B, the process repeats as aloop. Step 1 pulses a dot pattern on the surgical field. In Steps 2 and3, reflections (any light returning to a camera lens) are then mapped.In Steps 4 and 5, anatomical landmarks are identified in a wireframemodel of the surgical field and the dataset is fused with athree-dimensional solid model. In Step 6, the “fusion 3D model” thatresults is then used to generate virtual images for projection onto theeyepiece of the headset. In Step 7, the process then returns to a“START” (FIG. 16A) and begins again according to a clock speed and adata transmission rate. The process may be repeated at a speedsufficient to assist the surgeon and validate surgical intuition anddirect observation during the surgical procedure.

In advanced steps, surgical instrumentation may also be tracked, andgeometric calculations needed to orient and properly perform surgicalsteps may be shown in virtual tracks projected onto the virtual fusionimage, tracks that are visible only through the headset eyepiece.

FIG. 17 shows a block diagram of an example computing system 1700 bywhich at least aspects of surgical navigation may be implemented,arranged in accordance with at least some embodiments described herein.In this example, computing system 1700 includes a computing machine andperipherals dedicated for rapidly building detailed three-dimensionalvirtual models of a surgical field with an embedded three-dimensionalsolid model stored in a library and derived from 3D X-ray, CT, MRI orother imaging modality. Provision for representing a library of surgicalinstruments in the eyepiece is also provided. Similarly, prostheticimplant solid models may also be referenced from a database if required.

As mentioned, advantageously, the techniques described herein can beapplied to any device. It can be understood, therefore, that handheld,portable and other computing devices, systems, networks, and computingobjects of all kinds (including robotics) are contemplated for use inconnection with the various embodiments. Accordingly, the generalpurpose remote computer described schematically in FIG. 17 is but oneexample of a computing device.

FIG. 17 thus illustrates an example of a suitable computing systemenvironment in which one or more aspects of the embodiments describedherein can be implemented. Components of the computer machine mayinclude, but are not limited to, a processing unit, a system memory, anda system bus that couples various system components including the systemmemory to the processing unit.

Computers as shown typically include a variety of computer-readablemedia that can be any available media that can be accessed by computer.The system memory may include computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) and/orrandom access memory (RAM). By way of example, and not limitation,system memory may also include an operating system, applicationprograms, application program interfaces, other program modules, andprogram data, including databases and library data.

A user can enter commands and information into the computer throughinput devices termed here as “interfaces”. The eyepiece display moduleor other type of graphical display device may also be connected to thesystem bus via an interface, such as an output interface. In addition toa monitor, computers may also include other peripheral output devicessuch as speakers, which may be connected through an output interface. Amicrophone may also be included whereby the computer may respond toverbal commands. A haptic interface may also be utilized by the system.

The computer may operate in a networked or distributed environment usinglogical connections to one or more other remote computers, such asremote computer. The remote computer may be a workstation, a personalcomputer, a server, a router, a network PC, a peer device or othercommon network node, or any other remote media consumption ortransmission device, and may include any or all of the elementsdescribed above relative to the computer. The logical connectionsdepicted in FIG. 17 may include a network, such as a local area network(LAN) or a wide area network (WAN), but may also include othernetworks/buses. Such networking environments are commonplace in homes,offices, enterprise-wide computer networks, intranets and the Internetand are cited only as examples of the kinds of digital environments thatmay support the system.

There is little distinction left between hardware and softwareimplementations of aspects of systems; the use of hardware or softwareis generally (but not always, in that in certain contexts the choicebetween hardware and software can become significant) a design choicerepresenting cost vs. efficiency tradeoffs. There are various vehiclesby which processes and/or systems and/or other technologies describedherein can be effected (e.g., hardware, software, and/or firmware), andthat the preferred vehicle will vary with the context in which theprocesses and/or systems and/or other technologies are deployed. Forexample, if an implementer determines that speed and accuracy areparamount, the implementer may opt for a mainly hardware and/or firmwarevehicle; if flexibility is paramount, the implementer may opt for amainly software implementation; or, yet again alternatively, theimplementer may opt for some combination of hardware, software, and/orfirmware.

FIG. 18 shows an example headset view 1800 of a select command by whichat least aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. In thisexample, the headset view 1800 shows a surgical field 1805 with avirtual object 1810 projected in the eyepiece above the incision site.The gloved hand 1815 is used to manipulate the virtual object 1810, asshown in the following views. In this example, the surgeon's finger isused to select a vertebra from anatomical features visible through theincision. A virtual solid model of vertebra L3 is projected in theeyepiece using raytrace software so as to appear to hang above thesurgical field. Osseous features such as a fractured pedicle, and softtissue detail (such as a nerve pinch, disk bulge, and Doppler flow) maybe represented as needed to assist the surgeon using data fusion andsegmentation of multiple imaging inputs. Data from probes having sensoryoutputs, such as temperature, bone density, EKG, myelography, and EEGmay also be displayed in the headset eyepiece.

FIG. 19 shows an example headset view 1900 of a levitate command bywhich at least aspects of surgical navigation may be implemented,arranged in accordance with at least some embodiments described herein.In this example, the headset view 1900 shows a simple haptic command bywhich the surgeon can raise a virtual object 1905 to eye level forcloser inspection. A virtual vertebra is abstracted from a CT scan inthis example; the view of the surgeon's hand may be integrated from acamera in the headset. Structured light cast on the hand may besufficient to accurately detect and encode hand gestures by modeling thehands as a function of time in the viewing field. Radio reflective dotsor RFID chips may be included inside the gloves near the fingertips soas to make this process faster.

FIG. 20 shows an example headset view 2000 of a rotate command by whichat least aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. In thisexample, the headset view 2000 shows a haptic rotate command by which avertebral body 2005 may be instructed to rotate or turn over accordingto the motion of an index finger. When a suitable view is found, theuser may form a first to freeze the image.

FIG. 21 shows an example headset view 2100 of a zoom command by which atleast aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. In thisexample, the headset view 2100 shows a haptic zoom command, where thevirtual image 2105 may be positioned and expanded as displayed in thesurgeon's eyepiece. The enhanced size aids in detecting any pathologyand in visualizing angles and other dimensions. Data may be streamed tothe eyepiece for instant reference and may be updated in real time asthe surgeon indicates relevant features of interest.

FIG. 22 shows an example headset view 2200 of an angle measurementcommand by which at least aspects of surgical navigation may beimplemented, arranged in accordance with at least some embodimentsdescribed herein. In this example, the headset view 2200 shows anotherhaptic command, here a thumb and forefinger gesture indicating that thecomputer is to calculate and plot angles of relevant anatomical featuresof a vertebral body 2205. Streaming data may accompany the image. Analternate gesture may be a scissors motion, in which the surgeon pointsat a relevant feature and then scissors the index finger and middlefinger to indicate that angles are to be displayed.

FIG. 23 shows an example headset view 2300 of a slice command by whichat least aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. In thisexample, a graphical interface may be programmed to show the slice incross-section, and the slice may be repositioned by moving the hand upor down. Also shown are relevant dimensions, such as the thinnestcross-sectional dimension of the right and left pedicles, as tabulatedhere. While the surgeon may use an enlarged virtual image for finerdetail, the actual dimensions and angles reported may be as found invivo, vis a vis the patient's skeleton.

FIG. 24 shows an example headset view 2400 of surgical instrumentpositional analysis by which at least aspects of surgical navigation maybe implemented, arranged in accordance with at least some embodimentsdescribed herein. In this example, FIG. 24 shows that along withanatomical features, a database of surgical instruments, hardware, andprosthetics (implants) may also be shown in the eyepiece and manipulatedalong with positional analysis in real-time. Data may be streamed to theeyepiece to assist in accurate placement of surgical screws and K-wires,for example. These views may be supplemented by “in suite” views takenduring the surgery using a fluoroscope or other active imaging modality.AP and Lateral views are commonly used to guide and assess placement,but the system may supply active guidance and may integrate X-rays bybest-fit image analysis processes in order to update the solid model.Shown again here are radiobeacons useful in updating the wireframe modeland solid model fusion if needed, and for maintaining a live view invirtual mode when there is visual obstruction. Alternatively,instruments may be monitored and tracked using optical or electrometrictechnology as described in US Pat. Doc. Nos. 2005/0228270 and2013/0267833, which are hereby incorporated by reference.

Alternatively, a surgical plan may be used to generate a series ofreference views or a video, such that a guidepath for insertion of ascrew or an implant may be superimposed on the live view of the surgicalsite during the surgical procedure. Robotic steps may also be guided inthis way.

Importantly, by using high quality visual clues to simulate increasingdepth and intervening overlayers, the physician's intuitive sense ofposition and structure is enhanced, not interfered with. Deep structuresmay be shown, for example, in pale lime green shades that fade withdepth so as to be readily distinguished from the red and whitestructures of the exterior view. Vectored raytrace algorithms may beuseful for showing features in perspective to enhance a sense of depth.Special preference may be given to showing information by relevance, sothat distracting parts of the field are not shown. For example, theanatomy around a vertebral screw tract may be shown, but not thesuperjacent and subjacent vertebra, which could clutter the field. Inanother example, the anatomy of the tibia and fibula may be shown withan axial centerline when drilling a guide hole and shelf in preparationfor mounting a prosthetic on the tibial plafond. Deviations above andbelow, as well as medially and laterally from the centerline may behighlighted by projecting a comparative divergence angle between thetrue centerline and the projected drill centerline or blade cut. In thisway, costs for revision surgery may be reduced or avoided. In anotherexample, ligament balance and gait analysis may also be projected duringthe surgery to improve outcomes, but views of the metatarsals,navicular, and calcaneus may be hidden during surgery on the ankle jointunless summoned or contextually relevant to the procedure so as to avoidclutter. In another example, a virtual mirror may be summoned to showthe underside view when operating a saw that would otherwise obstructoversight of the cut.

FIGS. 25A, 25B and 25C are different views of surgical gloves by whichat least aspects of surgical navigation may be implemented, arranged inaccordance with at least some embodiments described herein. FIG. 25Ashows an example unaided view 2500 of surgical gloves. FIG. 25B shows anexample headset view 2505 of a visible surgical glove 2510 manipulatinga virtual object 2515. FIG. 25C shows an example headset view 2520 of aninvisible surgical glove manipulating a virtual object.

In these examples, the gloves may be visualized using a projected dotmatrix if desired. Alternatively, the gloves may be modified with aregistration stripe on each finger, and optionally an RFID antenna tagnear the fingernail of each finger and on the boney prominence of thescaphoid bone, medial and distal to the radius at the wrist so as toquickly convey a gesture command language. For more detail, the hands ofthe surgeon may be separately segmented from the remainder of the imageusing a blue, violet or ultraviolet dot matrix projector on the headsetin combination with a bandpass filter in the near visual and ultravioletrange. This segmented model may not automatically be incorporated intothe virtual augmentation presented to the eyepiece display, but may beused by the computer to track the motion of the hands for gesturerecognition and when needed for selectively making the hand visible inthe virtual view or invisible. In some instances, the hand may be anobstruction, so it may be desirable that the hand be visible to theunaided eye but not seen in the augmented eye. One or both hands may bemade invisible following digitization and corrective raytrace renderingof the underlying anatomy. By hand, the attached arm may also be subjectto selective display.

Relevant art includes U.S. Pat. Doc. No. 2002/0198472 to Kramer and U.S.Pat. No. 7,662,113 to Pearl. These patent documents are herebyincorporated by reference.

Similarly, surgical tools and prosthetics may be selectively segmentedmore rapidly by use of an alternate dot matrix. Motion dramaticallyincreases complexity of the calculation set required to render the handsand the tools, so the conical narrow beam of the headset may be moredense than a general pattern of dots covering the surgical field. Byprojecting a blue, violet or ultraviolet dot matrix to model thesurgeon's hands and tools, and an IR dot matrix to model the patient'sbody form and any surgically exposed bones or soft tissues,specialization of function dramatically increases the speed at whichcalculations can be made without interference or cross-talk. Anadvantage is that the segmented components can then be selectivelydisplayed in the eyepiece. For example, the hand may be holding a toolbut only the tool is shown so that the underlying anatomy is notobstructed. Surgeons can use their unaided eye to follow the tool, butcan also see the underlying solid model in the augmented eye and canfollow a projected guidepath to the target surgical site, such as ascrew hole through a pedicle of a vertebra or into the head of thehumerus, for example.

FIG. 26 shows a schematic representation of an example operating room2600 having projection systems by which at least aspects of surgicalnavigation may be implemented, arranged in accordance with at least someembodiments described herein. For example, a surgical suite with a morecomplex surgical navigation system is shown. Two dot-pattern projectionsystems, one that is ceiling mounted and another that is headsetmounted, are provided. In this view, the system includes hardware builtinto the headset 105 and the operating room 2600.

The headset 105 may include the following components: a dot matrixprojector (wavelength in blue or violet at fringe of visual range,optionally in UV range outside of UVA and UVB regions); bilateralcameras (2×2) with dual bandpass filter and CCD; radio receivers(right/center/left) for rough triangulation and image updating asheadpiece moves; eyepiece with display and lensing (termed here the“eyepiece”); eye pupil camera for fine adjustment of display ineyepiece; and fixed laser pointer.

The eyepiece may be configured to supply virtual images to one eye ofthe surgeon. The brain of the surgeon provides the needed “wetware” tointegrate these images into the corresponding view through the unaidedeye, if the images are correctly aligned. If needed, a camera may alsobe provided in the headset to monitor eye movements, pupil position andlens curvature of the unaided eye so as to project a biocompatiblevirtual display into the augmented eye, a virtual display that can beshared by both hemispheres of the brain and by the motor cortex andcerebellum without disorientation or confusion. Generally, this can beachieved by modeling of the perspective as seen by the unaided eye, andmodeling that perspective in virtual renderings using raytrace softwarefor the augmented eye display.

The operating room 2600 may include IR dot projectors 2605 from multiplesides of the ceiling (existing hardware) having at least three banks toproject a superimposed dot matrix from multiple angles, preferably at 60degree stations or at 90 degree stations relative to the surgical fieldso as to eliminate or reduce “blind” spots in the dot pattern.

Dot reflections may be captured by at least two cameras mounted on thesurgical headset and may be digitized for processing by an externalcomputer as presently conceived. A frame grabber synchronizes thecapture of images from each camera and the frames may be compared to mapa wireframe model of the surgical site. Reference landmarks may beidentified or may be taught to the computing machine by using a laserpointer or a finger plus voice commands. Once properly aligned, fusionsoftware may be used to orient a solid model of the patient anatomy ontothe wireframe model of the surgical field. Segmented solid modellibraries may contain individual bones, organs, or soft tissue elements.The libraries may also include surgical tools and libraries ofprosthetics and surgical hardware.

Surgical tools may include at least two radio antennae wrapped around astem of the tool so as to respond omnidirectionally to a radioexcitation and to emit a signature radio signal in response toexcitation. RFID structures, for example, may be integrated into (orattached to) the tools. Similar radiobeacons may be integrated into (orattached to) prostheses, and as described above, may also be integratedinto or inserted under the surgeon's gloves.

Also shown is an operating table 2610 that may serve as a stablecoordinate reference frame for the headset. Four radiobeacons may beplaced at the corners of the table so as to reflect radio signaturesback to a triplet array of radio receivers in the headset. Simpletriangulation algorithms permit a computing machine to calculate headsetposition from time of flight measurements performed with a highfrequency clock. Frequencies of 5.5 MHz or higher may be used to improvethe accuracy of the positional sensing.

Advantageously, the radiobeacons may also be placed at the corners of aMayo table, a slip-on cover on the Mayo table, the corners of theoperating table, or a mat under the surgeon's feet, each corner having aradio reflective antenna equipped with an identifiable signaturereflection. In this way, the headset orientation may be tracked by anexternal reference frame, but one that is not subject to the weaknessesof optical tracking. The surgeon may calibrate the system by pointingout at least one beacon associated with a boney prominence or obviousanatomical feature that is present on the wireframe map, and theinternal solid model and the rest of the beacons can then be formed intoa spatial map that is determinate and inflexible for the duration of theprocedure. If the patient is rolled over, for example, only one or twobeacons may be disturbed, so their positions may be refreshed while theremaining beacons are fixed in a master coordinate array stored incomputer memory. Tracking the headset may be relatively easy usingstandard matrix algebra and may require substantially less computationalpower.

At least one computer/server may be connected to the headset, generallyby a data umbilicus as shown in FIG. 2. The umbilicus may also routefiber optics to the headset dot matrix projector. The computer/servermay include client software routines and memory sufficient to handle andstore image-related digital databases.

The following is a partial list of software engines that may be used inthe surgical navigation systems: a subroutine for constructing awireframe model of a surgical field and roughing in relative viewpointof a headset; a subroutine for creating a 3D model from tomographicaldatasets; a subroutine for administering a data library of 3D models,including tools; a subroutine for registering the 3D model with theactual patient; a subroutine for “segmenting” anatomy (splitting animage into its component body elements and treating each body elementseparately in the model); a subroutine for tracking tools; a subroutinefor tracking an eyeball of an unaided eye; a subroutine for tracking asurgeon's hands (likely with special gloves); a subroutine for imageanalysis, including geometry on segmented anatomical elements; asubroutine for handling voice commands; and a subroutine for dynamicraytrace of virtual display elements with perspective (dynamicindicating updating at a frame rate of at least 30 fps, or inreal-time).

Reflections of structured or unstructured light dots may be used totrack the surgical field and anatomical landmarks (no retroreflectiveball apparatus is used) and reflections of VIS dots (here indicating ablue, violet or ultraviolet dot matrix) may be used to track the handsand tools. The radiobeacons may be coarse navigational aids for roughingin the position of tools and a permanent reference on the outsideboundary of the surgical field (so as to size the area where finemapping is needed and to speed up the recapture of the correctorientation frame when the visual image is interrupted). By using twodot systems that are superimposed, the gloves can be subtracted from thesurgical view or vice versa, and blocking can be selectively used tobetter reinforce the illusion of stereopsis without a binary eye system.Alternatively, body and most relevant bones or anatomical targets can bedisplayed as a solid and hands or tools can be displayed as a ghostimage, or vice versa. Tools may be displayed without the handssupporting them so as to minimize obstruction of the surgical field.Tools may also be provided with analytics, including angle and guidepathprojections.

Also a component in the system is “wetware” which the operating surgeonis proficient in using. The inputs to the wetware are optimized to avoidfatigue and to prevent confusion and defects in hand-eye coordinationthat could result from misaligned virtual augmentation. A functionalcorpus callosum and optic chiasma are needed to interpret the unaidedeye view in conjunction with the augmented views in the eyepiece. Thisis referred to as “wetware”. So the system may be interpreted to have asoftware component, a hardware component, and a wetware component.Generally one unaided eye is used and the augmented eye is provided withan eyepiece for receiving virtually enhanced images and data feeds suchas text. Voice commands and haptic gestures may be used to control thevirtual display and may be used to turn off a view of one or the otherhand, for example, so as to disable ambiguous visual cues such as ananatomical view superimposed on top of the surgeon's arm or wrist. Whenused in combination with a segmented library of anatomical parts, toolsand prosthetics, the capacity to also segment the surgeon's hands offermultiple advantages in reducing image clutter, improving depth cues, anddirecting computing operations without interference from backgroundnoise and without the need for remote control interfaces.Advantageously, segmentation also permits presentation of anatomicalviews with reduced complexity and clutter, limiting the view to the morerelevant structures. This again reduces inappropriate superimposing ofimages and simplifies the computational process.

FIG. 27 shows an example headset view 2700 of multiple virtual objectsby which at least aspects of surgical navigation may be implemented,arranged in accordance with at least some embodiments described herein.For example, while one vertebra is shown in FIGS. 25B and 25C, multiplebone clusters 2705 may also be manipulated where surgically relevant, asshown in FIG. 27. A display of all the vertebra would result in visualclutter, but showing two contacting vertebrae may be relevant to a rangeof spinal surgeries, such as discectomy. A segmented image library isutilized in this example, and FIG. 27 illustrates how informationrelevant to the surgical procedure may be presented with clarity byelimination of unwanted elements such as ribs and uninvolved vertebrae.This image may also be less computationally dense and can be morerapidly updated as the surgeon's view changes. In this example, thesurgeon may be looking through the thorax and views the ventral aspectof the spinal cord while pointing to particular vertebra with a pair ofsurgical gloves that are visible in the unaided eye and are used invirtual space to select the anatomy of interest. Here, finger pointingmay be used to pick out two vertebrae having a normal joint space. Asdescribed above, further manipulation of the image may result in astreaming display providing quantitative measurement of the discthickness. Soft tissue views may also be superimposed if desired and theimage may be accompanied by nerve transmission studies or sectionsshowing myelographs if available.

The surgeon's fingertip may also trace the spine to identify spinousprocesses. This method of identifying anatomical landmarks is termed“palpation”, and may be digitally enhanced as described below.

FIGS. 28A and 28B show a schematic representation of an examplefingertip portion 2800 of a surgical glove by which at least aspects ofsurgical navigation may be implemented, arranged in accordance with atleast some embodiments described herein. In this example, surgicalgloves may be modified to include a cluster of IC chips and supportingcircuitry as needed to sense touch. Computer-augmented palpation may beachieved by combining a cluster or at least one radiobeacon in afingertip so that location may be tracked and one or more ICs forspecial functions. Shown here by way of example is an MCU ARM chip fordigitizing the sensation of touch, four radiobeacons, a 6-axisaccelerator/gyroscope and magnetometer integrated circuit withthermistor, and a piezoelectric chip for delivering ultrasound whencontacted with a substrate such as skin.

FIG. 28B shows an example side view of the IC cluster and radiobeaconswith fractal antennas. Other antenna configurations, such as dielectricor ceramic microstrip antennas and dipole antennas, while not limitedthereto, may be used to limit the size and complexity of the antennastructure.

FIG. 29 shows an example radiobeacon reference coordinate frame andheadset by which at least aspects of surgical navigation may beimplemented, arranged in accordance with at least some embodimentsdescribed herein. In this example, the radiobeacon reference coordinateframe and eyewear are configured to project views of the internalanatomy and data to the surgeon's left eye. The reference coordinateframe may be used to fuse larger 3D anatomical images such as a CT modelto points designated by the surgeon. In this more field-like operatingtheater, the surgeon may drag and drop the CT model into the correctanatomy using a pre-haptic set of gestures and haptic features builtinto the gloves. This may be achieved without a dotted light projectorand may include at least one radiobeacon mounted in each of a pair ofsurgical gloves and a plurality of radio receivers mounted inassociation with headgear worn by the surgeon, the radio receiversfunctioning to perform triangulation on the radiobeacons dispersed inthe surgical napkin on the Mayo table and on the corners of the gurney.These may provide the foundational reference coordinate system used topresent optically realistic augmented reality displays in the eyepiece.

In a simplified method, radiobeacons may be used as the primaryreference frame for data fusion with more complex datasets. The eyepiecemay display virtual views in a correct anatomical orientation as seen bythe surgeon and move the perspective as the surgeon moves. In thisexample, a radiobeacon reference frame may be used as a foundation foraugmented reality presented via a surgical eyepiece, heads-up display,or pseudo-holographic view.

As noted above, radiobeacons may be used to create reference frames forfusion of spatial 3D datasets with patient anatomy as visible to asurgeon in alternation with or to complement with an optical scanningsystem. By correlating a 3D model with a reference frame and associatingthat reference frame with an accurate positioning of the patient, the 3Dmodel may be projected as a virtual image into an eyepiece of a headsetsuch that the 3D model is closely aligned with the actual anatomy. Theworking example is that of a CT dataset, which when superimposed in avirtual view on the patient, reveals underlying boney anatomy notdirectly visible to the surgeon prior to dissection. The basic referencecoordinate frame may be provided by a set of radiobeacons disposed inthe operating theater.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies regardless of the particular type of signal bearing medium usedto actually carry out the distribution. Examples of a signal bearingmedium include, but are not limited to, the following: a recordable typemedium such as a floppy disk; a hard disk drive, a CD, a DVD, a digitaltape, a computer memory, etc.; and a transmission type medium such as adigital and/or an analog communication medium (e.g., a fiber opticcable, a waveguide, a wired communications link, a wirelesscommunication link, etc.).

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors (e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities). A typical data processingsystem may be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely examples, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations. In addition, even if a specificnumber of an introduced claim recitation is explicitly recited, thoseskilled in the art will recognize that such recitation should beinterpreted to mean at least the recited number (e.g., the barerecitation of “two recitations,” without other modifiers, means at leasttwo recitations, or two or more recitations). Furthermore, in thoseinstances where a convention analogous to “at least one of A, B, and C,etc.” is used, in general such a construction is intended in the senseone having skill in the art would understand the convention (e.g., “asystem having at least one of A, B, and C” would include but not belimited to systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.). In those instances where a convention analogous to “atleast one of A, B, or C, etc.” is used, in general such a constructionis intended in the sense one having skill in the art would understandthe convention (e.g., “a system having at least one of A, B, or C” wouldinclude but not be limited to systems that have A alone, B alone, Calone, A and B together, A and C together, B and C together, and/or A,B, and C together, etc.). It will be further understood by those withinthe art that virtually any disjunctive word and/or phrase presenting twoor more alternative terms, whether in the description, claims, ordrawings, should be understood to contemplate the possibilities ofincluding one of the terms, either of the terms, or both terms. Forexample, the phrase “A or B” will be understood to include thepossibilities of “A” or “B” or “A and B.”

In addition, where features or aspects of the disclosure are describedin terms of Markush groups, those skilled in the art will recognize thatthe disclosure is also thereby described in terms of any individualmember or subgroup of members of the Markush group.

As will be understood by one skilled in the art, for any and allpurposes, such as in terms of providing a written description, allranges disclosed herein also encompass any and all possible subrangesand combinations of subranges thereof. Any listed range can be easilyrecognized as sufficiently describing and enabling the same range beingbroken down into at least equal halves, thirds, quarters, fifths,tenths, etc. As a non-limiting example, each range discussed herein canbe readily broken down into a lower third, middle third and upper third,etc. As will also be understood by one skilled in the art all languagesuch as “up to,” “at least,” and the like include the number recited andrefer to ranges which can be subsequently broken down into subranges asdiscussed above. Finally, as will be understood by one skilled in theart, a range includes each individual member. Thus, for example, a grouphaving 1-3 cells refers to groups having 1, 2, or 3 cells. Similarly, agroup having 1-5 cells refers to groups having 1, 2, 3, 4, or 5 cells,and so forth.

From the foregoing, it will be appreciated that various embodiments ofthe present disclosure have been described herein for purposes ofillustration, and that various modifications may be made withoutdeparting from the scope and spirit of the present disclosure.Accordingly, the various embodiments disclosed herein are not intendedto be limiting, with the true scope and spirit being indicated by thefollowing claims.

1. A method, comprising: receiving, by a computing system, an externalthree-dimensional model of a surgical site from the viewpoint of aheadset, wherein the external three-dimensional model is derived fromreflected light; aligning, by the computing system, the externalthree-dimensional model with an internal three-dimensional model of thesurgical site from the viewpoint of the headset, wherein the internalthree-dimensional model is derived from medical imaging, and generatingan aligned view; providing, by the computing system, the aligned view tothe headset; and updating, by the computing system, the aligned view inreal-time while the headset is moved or the surgical site is moved ormodified during a surgical procedure.
 2. The method of claim 1, whereinthe reflected light is produced from an array of light beams projectedonto the surgical site.
 3. The method of claim 2, wherein the array oflight beams is projected from the headset.
 4. The method of claim 2,wherein the array of light beams comprises infrared light.
 5. The methodof claim 1, wherein the external three-dimensional model comprises awireframe model of the surgical site.
 6. The method of claim 1, whereinthe medical imaging includes at least one of the following: computerizedtomography (CT) scanning, magnetic resonance imaging (MRI), x-rayimaging, or ultrasound imaging.
 7. The method of claim 1, wherein theinternal three-dimensional model is segmented according to relevantanatomy therein, thereby deriving a library of segment-relevantanatomical elements.
 8. The method of claim 7, wherein the aligned viewshows segment-relevant anatomical elements In an anatomically correctcorrespondence with the surgical site.
 9. The method of claim 1, furthercomprising: receiving, by the computing system, a headset position andorientation relative to the surgical site.
 10. The method of claim 9,wherein the headset position and orientation is derived from radiotracking that utilizes at least one fixed radiobeacon to provide acoordinate reference frame for the headset.
 11. The method of claim 9,wherein the headset position and orientation is derived from opticaltracking that utilizes at least one fixed optical beacon to provide acoordinate reference frame for the headset.
 12. The method of claim 9,wherein the headset position and orientation is derived from inertialguidance provided by the headset having an accelerometer.
 13. The methodof claim 9, wherein updating the aligned view in real-time comprisesaligning the headset position and orientation with the internalthree-dimensional model.
 14. The method of claim 1, further comprising:receiving, by the computing system, a hand gesture relative to thesurgical site.
 15. The method of claim 14, wherein the hand gesture isderived from radio tracking that utilizes at least one radio-reflectivepatch or radio-frequency identification (RFID) chip inside a glove. 16.The method of claim 14, wherein the hand gesture is derived from opticaltracking that utilizes structured light projected onto a glove.
 17. Themethod of claim 14, further comprising: manipulating, by the computingsystem, an anatomical element in the internal three-dimensional model asa function of the received hand gesture; and updating, by the computingsystem, the aligned view in real-time to show the manipulation of theanatomical element.
 18. The method of claim 17, wherein manipulating theanatomical element includes at least one of the following: isolating,levitating, rotating, magnifying, cross-sectioning, or measuring. 19.The method of claim 1, further comprising: receiving, by the computingsystem, a surgical instrument position and orientation relative to thesurgical site, or a sensor output of the surgical instrument.
 20. Themethod of claim 19, wherein the instrument position and orientation isderived from radio tracking that utilizes at least one radio-reflectivepatch or radio-frequency identification (RFID) chip on the instrument.21. The method of claim 19, further comprising: updating, by thecomputing system, the aligned view in real-time to show the instrumentposition and orientation relative to the internal three-dimensionalmodel, or the sensor output of the surgical instrument.
 22. The methodof claim 21, wherein the instrument position or orientation or sensoroutput includes at least one of the following: depth, angle, relativeangle, relative elevation, volume, temperature, pressure, oxygenation orenervation.
 23. An article of manufacture including a non-transitorycomputer-readable medium having instructions stored thereon that, inresponse to execution by a computer system, cause the computer system toperform operations comprising: receiving an external three-dimensionalmodel of a surgical site from the viewpoint of a headset, wherein theexternal three-dimensional model is derived from reflected light;aligning the external three-dimensional model with an internalthree-dimensional model of the surgical site from the viewpoint of theheadset, wherein the internal three-dimensional model is derived frommedical imaging, and generating an aligned view; providing the alignedview to the headset; and updating the aligned view in real-time whilethe headset is moved or the surgical site is moved or modified during asurgical procedure.
 24. A system, comprising: a processor; and anon-transitory memory having instructions stored thereon that, inresponse to execution by the processor, cause the system to performoperations comprising: receiving an external three-dimensional model ofa surgical site from the viewpoint of a headset, wherein the externalthree-dimensional model is derived from reflected light; aligning theexternal three-dimensional model with an Internal three-dimensionalmodel of the surgical site from the viewpoint of the headset, whereinthe internal three-dimensional model is derived from medical imaging,and generating an aligned view; providing the aligned view to theheadset; and updating the aligned view in real-time while the headset ismoved or the surgical site is moved or modified during a surgicalprocedure.
 25. The method of claim 1, further comprising: receiving, bythe computing system, a hand gesture relative to the surgical site. 26.The method of claim 25, wherein the hand gesture is derived from radiotracking that utilizes at least one radio-reflective patch orradio-frequency identification (RFID) chip inside a surgical glove. 27.The method of claim 25, further comprising: manipulating, by thecomputing system, the aligned view in real-time according to thereceived hand gesture.
 28. The method of claim 27, wherein the handgesture includes at least one of the following: a select command, alevitate command, a rotate command, a stop command, a zoom command, ameasure command, or a slice command.